Pandas Data Analysis Overview Master data analysis with Pandas, the powerful Python library for data manipulation and analysis. Learn to clean, transform, analyze, and visualize data effectively. Learning Objectives Load and manipulate data from various sources (CSV, Excel, SQL, APIs) Clean and transform messy datasets Perform exploratory data analysis (EDA) Aggregate and group data for insights Create compelling visualizations Optimize performance for large datasets Core Topics 1. Pandas DataFrames & Series Creating DataFrames from various sources Indexing and selecting data (loc, iloc, at, iat) Filtering and boolean indexing Adding/removing columns and rows Data types and conversions Code Example: import pandas as pd import numpy as np

Create DataFrame

data

{ 'name' : [ 'Alice' , 'Bob' , 'Charlie' , 'David' ] , 'age' : [ 25 , 30 , 35 , 28 ] , 'salary' : [ 50000 , 60000 , 75000 , 55000 ] , 'department' : [ 'IT' , 'HR' , 'IT' , 'Sales' ] } df = pd . DataFrame ( data )

Indexing and filtering

it_employees

df [ df [ 'department' ] == 'IT' ] high_earners = df . loc [ df [ 'salary' ]

55000 , [ 'name' , 'salary' ] ]

Adding calculated columns

df [ 'annual_bonus' ] = df [ 'salary' ] * 0.10 df [ 'age_group' ] = pd . cut ( df [ 'age' ] , bins = [ 0 , 30 , 40 , 100 ] , labels = [ 'Young' , 'Mid' , 'Senior' ] ) print ( df ) 2. Data Cleaning & Transformation Handling missing data (dropna, fillna, interpolate) Removing duplicates String operations and text cleaning Date/time parsing and manipulation Type conversions and casting Applying custom functions (apply, map, applymap) Code Example: import pandas as pd

Load data with missing values

df

pd . read_csv ( 'sales_data.csv' )

Handle missing values

df [ 'price' ] . fillna ( df [ 'price' ] . median ( ) , inplace = True ) df [ 'category' ] . fillna ( 'Unknown' , inplace = True ) df . dropna ( subset = [ 'customer_id' ] , inplace = True )

Clean text data

df [ 'product_name' ] = df [ 'product_name' ] . str . strip ( ) . str . lower ( ) df [ 'product_name' ] = df [ 'product_name' ] . str . replace ( '[^a-zA-Z0-9 ]' , '' , regex = True )

Convert dates

df [ 'order_date' ] = pd . to_datetime ( df [ 'order_date' ] ) df [ 'year' ] = df [ 'order_date' ] . dt . year df [ 'month' ] = df [ 'order_date' ] . dt . month

Remove duplicates

df . drop_duplicates ( subset = [ 'order_id' ] , keep = 'first' , inplace = True )

Apply custom function

def categorize_price ( price ) : if price < 50 : return 'Low' elif price < 100 : return 'Medium' else : return 'High' df [ 'price_category' ] = df [ 'price' ] . apply ( categorize_price ) 3. Aggregation & Grouping GroupBy operations Aggregation functions (sum, mean, count, etc.) Pivot tables and cross-tabulation Multi-level indexing Window functions (rolling, expanding) Code Example: import pandas as pd

Sample sales data

df

pd . read_csv ( 'sales.csv' )

GroupBy aggregation

dept_stats

df . groupby ( 'department' ) . agg ( { 'salary' : [ 'mean' , 'min' , 'max' ] , 'employee_id' : 'count' } )

Multiple groupby

sales_by_region_product

df . groupby ( [ 'region' , 'product_category' ] ) [ 'sales' ] . sum ( )

Pivot table

pivot

df . pivot_table ( values = 'sales' , index = 'product_category' , columns = 'quarter' , aggfunc = 'sum' , fill_value = 0 )

Rolling window (moving average)

df [ 'sales_ma_7d' ] = df . groupby ( 'product_id' ) [ 'sales' ] . transform ( lambda x : x . rolling ( window = 7 , min_periods = 1 ) . mean ( ) )

Cumulative sum

df [ 'cumulative_sales' ] = df . groupby ( 'product_id' ) [ 'sales' ] . cumsum ( ) 4. Data Visualization Matplotlib basics Seaborn for statistical plots Pandas built-in plotting Customizing plots Creating dashboards Code Example: import pandas as pd import matplotlib . pyplot as plt import seaborn as sns

Set style

sns . set_style ( 'whitegrid' )

Load data

df

pd . read_csv ( 'sales_data.csv' )

1. Line plot - Sales trend over time

df . groupby ( 'month' ) [ 'sales' ] . sum ( ) . plot ( kind = 'line' , figsize = ( 10 , 6 ) ) plt . title ( 'Monthly Sales Trend' ) plt . xlabel ( 'Month' ) plt . ylabel ( 'Total Sales ($)' ) plt . show ( )

2. Bar plot - Sales by category

category_sales

df . groupby ( 'category' ) [ 'sales' ] . sum ( ) . sort_values ( ascending = False ) category_sales . plot ( kind = 'bar' , figsize = ( 10 , 6 ) ) plt . title ( 'Sales by Category' ) plt . xlabel ( 'Category' ) plt . ylabel ( 'Total Sales ($)' ) plt . xticks ( rotation = 45 ) plt . show ( )

3. Histogram - Price distribution

df [ 'price' ] . hist ( bins = 30 , figsize = ( 10 , 6 ) ) plt . title ( 'Price Distribution' ) plt . xlabel ( 'Price ($)' ) plt . ylabel ( 'Frequency' ) plt . show ( )

4. Box plot - Salary by department

df . boxplot ( column = 'salary' , by = 'department' , figsize = ( 10 , 6 ) ) plt . title ( 'Salary Distribution by Department' ) plt . suptitle ( '' ) plt . show ( )

5. Heatmap - Correlation matrix

corr

df [ [ 'age' , 'salary' , 'years_experience' ] ] . corr ( ) sns . heatmap ( corr , annot = True , cmap = 'coolwarm' , center = 0 ) plt . title ( 'Correlation Matrix' ) plt . show ( ) Hands-On Practice Project 1: Customer Analytics Analyze customer purchase behavior and segmentation. Requirements: Load customer transaction data Clean and prepare dataset Calculate RFM (Recency, Frequency, Monetary) metrics Customer segmentation Visualize insights Generate executive summary Key Skills: Data cleaning, aggregation, visualization Project 2: Time Series Analysis Analyze sales trends and forecast future performance. Requirements: Load time series data Handle missing dates Calculate moving averages Identify trends and seasonality Detect anomalies Create interactive visualizations Key Skills: Time series operations, rolling windows, plotting Project 3: Data Quality Report Build automated data quality assessment tool. Requirements: Check for missing values Identify duplicates Detect outliers Validate data types Generate quality metrics Export HTML report Key Skills: Data validation, statistical analysis, reporting Assessment Criteria Load and clean real-world datasets efficiently Perform complex data transformations Use GroupBy for aggregations Create insightful visualizations Handle missing and inconsistent data Optimize performance for large datasets Document analysis with clear explanations Resources Official Documentation Pandas Docs - Official documentation NumPy Docs - NumPy documentation Matplotlib Docs - Plotting library Learning Platforms Kaggle - Free Pandas course DataCamp - Interactive courses Python for Data Analysis - Wes McKinney's book Tools Jupyter Notebook - Interactive development Google Colab - Cloud notebooks Anaconda - Data science distribution Next Steps After mastering Pandas, explore: Scikit-learn - Machine learning SQL - Database querying Apache Spark - Big data processing Tableau/Power BI - Business intelligence tools